Audio-Driven Dubbing for User Generated Contents via Style-Aware Semi-Parametric Synthesis

Existing automated dubbing methods are usually designed for Professionally Generated Content (PGC) production, which requires massive training data and time to learn a person-specific audio-video mapping. In this paper, we investigate an audio-driven method that is more feasible User (UGC) production. There two unique challenges design UGC: 1) the appearances of speakers diverse arbitrary as needs generalize across users; 2) available video one speaker very limited. order tackle above challenges, first introduce new Style Translation Network integrate speaking style target content source via cross-modal AdaIN module. It enables our model quickly adapt speaker. Then, further develop semi-parametric renderer, takes full advantage limited unseen video-level retrieve-warp-refine pipeline. Finally, propose temporal regularization generating continuous videos. Extensive experiments show generates videos accurately preserve various styles, yet with considerably lower amount in comparison existing methods. Besides, achieves faster testing speed than most recent